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Criteria for the Selection of Covariates 
in Nonrandomized Designs 

Alan C. Bugbee, Jr. 
• University, of Pittsburgh, "1983 

» it 

( ABSTRACT 

♦ f 

This paper discusses four criteria for selecting covariat.es 
when utilizing^ the analysis of covariance in , quasi- 



experimental 



research : 



Completeness , 



Causatipn , 



Independence, and Fallibility. Completeness is the. extent 

to which the known or suspected Sources, of bias are 

accounted for by covariates used in an experiment. 
C* - 

Causation \is the requirement' that, covariates used in 
nonrancfom^ese&rch designs are ' causally; related to the 
dependent variable- of a study. 'independence is the 



requirement 
interaction . 



that thepe . is no treatm'ent/covariate 
Fallibility refers to the reliability of the 
covarlate measure. Ways of determining to what degree these 
criteria are met are suggested. 

Keywords:- Analysis of Covariance 4 (ANCOV^) : Quasi-Exper iment ; 
Nonrandomized Design; Covariate Selection. . ' 
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I like to classify the criteria for the selection of 
covariates in quasi-experimental research using the analysis 
, of covariance into four categories : Completeness , 

Causation, Independence, and Falibility. The^first two 
criteria concern the association of f a covar^ate. with the N 
dependent variable. TJie third criterion, Independerfce , 
concerns the relation of the covariate with the treatment of 

m 

an 'experiment . The fourth criterion, Falikility, concerns 
the .accuracy of measurement of the covfr'iate. 

The purpose of. the use of ANCOVA in quasi-experimental 
research is to remove bias from study, to statistically 
adjust for pre-treatment differences between treatment 
groups in order to eliminate rival hypotheses that 
post-tr eatuvent differences in a study are 'functions qf 
factors -other than treatment. To accomplish this, 

covariates utilized / in .'a research study, must be' causally 
related to the dependent variable. This is the criterion bf 
oausatlon. In other words, to make ad justment f or bias, the 
covariates used must be sources of bias. This is an 
essential ^ difference between the use of ANCOVA in 
quasi-exper itnental/ research gnd in true experimental 
research. 1 In true experimentation using ANCOVA, as H. F. 
Smith (1957) has "argued, it does not matter whether the 



\ 
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covariate is causally related to the dependent variable. 

What does matter^is the strength of the' relation between the 

.dependent variable and the covariate. The works of Cox 

(1957) and FeTdt (1958) address the importance of the degree 

of this strength of relation for the'use of ANCOVA in true 

experimentation. 

In order, to, remove bias and eliminate rival hypotheses, 

the researcher mu^fc identify the .known sources of bias and 

util ize these sources as covar iates in an analysis . This is 

• ♦ 

the crit-erion of completeness. It is only when these 
sources of bias are adjusted in, the analysis that a study 
may be considered to be unbiased. In the words of Cook -arid 
Campbell, . $ 

When it appears that all plausible biases have t>esn 
taken into account- and a treatment emerges in spit/e 
of them all, conclusions can be made with reasonable 
confidence. (Cook & Campbell, 1979, p. 201) « -s 

The emphasis of this criterion is on /£nown and plausible 
sources pf bias. ,When research involves human subjects, the 
possible. sources of bias may be many. For thi$ • reason, a 
researcher planning^ to use ANCOVA in quasi-exper imentat ioru 
must be well-^versed i(( previous research on tj/e experiment 
of interest and in both the subjects and treatments to. be 
used 1 . It iV->with this <ual knowledge that known and 



* 

plausiliJLe sources of bias can be accounted for 'and utilized, 
and, in this manner f the Conclusions of a study may be 
considered to be relatively unbiased. 

r 

The criterion of causation is ' met by incorporating * as 
* 

covariates only those variables that derive from a 

convincing causal moctel for the dependent variable. The 

researcher should report that the results of a study are 

conditional upon the sources of bias that were removed, and, 

« 

therefore, the conclusions of a study are also conditional 
upon this adjustment. 

j The 'third criterion, Independence, refers^ to" a lack of 
interaction between the covariates and the treatment of a 
study. The importance of this criterion is not so* much, for 
its effect oil the* ANCOVA model and statistical technique, 
but rather for its effects on the feasibility of conclusions 

4 

of a study using* ANCOVA. That is, "...the covarriate being 

r * 

t t 

unaffected by the treatment is of great importance ... if 
the interpretation of the adjusted treatment means is to be 
mwrningfCM.'" (Baker, 1D72„p.32). - When there is 
covar iate/ treatment /- interaction,^ the regression adjustment 
on the dependent variable, "...may remove * part of the 
treatment affect or produce spurious treatment affect." 
(Elashoff, 1969,p-38'8) 



A way to test this independence was proposed by Huitema in 
1980. He suggests that an analysis of variance be performed 
on covariates across" groups under study. His rationale for 
this^is that, 



If treatments have not been applied but the ANOVA F 
is significant -this is a warning that the ANCOVA F 
and adjusted means are almost certain to be biased 
as reflects the treatment effects. If the ANOVA on 
the covariates is not significant , the ANCOVA F and 
adjusted means are still subject to problems of bias 
but the degree of. bias, although unknown, may not be 
• large. (Huitema, 1980, p. 109)/ » % 

Although this test does not provide. 6 definitive answer to 
whether or not the covariates and treatment are independent, 
it does provide useful information about this criterion. It 
is recommended that this test be routinely performed as part 
of an analysis of covariance'. * 

Another way to test this, particularly when a researcher 
wishes to test the feasibility of the use' of a variable 
measured either during or after the application ' of 
treatment, ie. a cross-sectional covariate, is to analyze a 
post- treatment covariate by a pre- treatment covariate • 
Because a oross -sectional' covariate is assumed- to 
independent of treatment, it seems* reasonable that two 
measures of thi^ 4 covariate, one pre-treatment , one 
post-treatment, would^not show a significant difference fr$m 
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one another. A possible way to test this, is to perform an 
^analjsis of covariance on the post-treatment variable* 
covarying - on the pre-treatment or longitudinal measure of 
the same variable. It would be expected that the ANCOVA F 
will not be significant. If this" was not the case, it* would 
indicate that the cross-sectional covariate, wh,ich -was 
assumed to be statistically independent of treatment, was 
affected by treatment £nd that the results of a study using 
this covariate would be biased. This'is'a test' of assumed 
independence, of covariates which ar.e measured during or 
" after the application of treatment and may prove useful to 
researchers who are considering the utilization of a 

v 

cross-sectional covariate* in their research. To be sure, it 
requires extra^time an<| work, but, it may be of value a* a 
justification- for thg use ,.of a covariate which is measured 
.during or after the application o # f treatment. ^ 

The. fourth criterion, Falibility, refers to how well the 
covari&te utilized in a §tudy* is measured without error; thfe 
r&lia-bility * of the co\rariate measure.. This criterion comes 
-from the assumption of the ANCOVA model that, " ....the 
^covariate is a fioced mathematical Variable measured without 
error*, ■ not d , stochastic variable.. 11 (Wildt & Ahtola , 
1978, p. 89) In quasi-experimental Resigns where subjects are 



not randomly assigned to treatment , (the fallibility of the 
oovariate measure can -be of particular \importance because, 
".;.if the covariate means are not \mial , the difference 
between the adjusted means is p^rftly a ^function of t$* 
reliability .of the covariate ." (Huitema, 1980 , p. 1 13). 
While in randomized designs the affect of unreliability is 
to* lower the precision of the F test, In nonrandomized or 
quasi-experimental designs, the degree of$ f alibility of the 
covariate may affect the meaning and inter pretabil ity of the 
results of an analysis. % That is, adjusted mean differences 
could be a reflection of the reliability of the covariate 
measure rather than adjustments for bias, and consequently 
the results of the analysis could *be misrepresentations. 
There are two suggestions regarding this criterion.* First, 



a researcher should utilize the* most reliable available 

f 

apparati in the measurement of cOvariates. Second, a 
researcher should check the reliability of the instrument 
after its use in a study. * Although this latter point adds a 
burden to a quasi-experimental study using ANCOVA, with the 
present availability of statistical computer packages, and 
considering the importance of this criterion on the meaning * 
of results, the slight inconvenience of this procedure is 
well worth the . time it takes to conduct . 



As Glass, Peckham, and Sanders pointed out, the 
assumptions of a statistical model, "...are always false to 
a greater or lesser degree. **( 1972, p. 237) This is also true 
for meeting these criteria for covariate selection in 
quasi-experimental designs ; the criteria are never entirely 
met, they are only collectively achieved by dfegree. 
Researchers should strive to meet/ all of these criteria but , 
^Ln lieu .of that,' they should report the degree to which 
criteria are met and take this into account -when 
interpreting and reporting results and ^mea^^ of "a study. 
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